Information processing method, information processing device, and non-transitory computer readable storage medium

ABSTRACT

An information processing method includes: for each of one or more users, acquiring image data including an eye of each of the users; detecting eye gaze information indicating an eye gaze of each of the users based on information indicating the eye of each of the users included in the image data; performing personal authentication on each of the users based on information indicating the eye of each of the users included in the image data; acquiring personal information for identifying each of the users for which the personal authentication has been performed; generating management information in which the personal information of the one or more users and the eye gaze information of the one or more users are associated with each other; and outputting the management information.

TECHNICAL FIELD

The present disclosure relates to a technique of generating informationin which personal information of a user and information indicating aneye gaze of the user are associated.

BACKGROUND ART

The eye gaze detection technique is used in various applications such asestimation of a person's interest target, estimation of a person's statesuch as drowsiness, and a user interface that performs input toequipment by an eye gaze. When estimating the state and behavior of aperson based on eye gaze information, it is useful to use information inwhich the eye gaze information and information regarding the person areassociated with each other. As such an example, Patent Literature 1discloses a technique of using, when estimating a behavior of a customerin a store, information in which eye gaze information of the customer inthe store is associated with attribute information of the customer suchas age and gender and information (Point Of Sales (POS) information)regarding a product purchased by the customer.

However, in the technique disclosed in Patent Literature 1, theequipment becomes large in scale, and it is difficult to accuratelyassociate eye gaze information with information regarding the person,and hence further improvement is necessary.

CITATION LIST Patent Literature

-   Patent Literature 1: JP 2017-102564 A

SUMMARY OF INVENTION

The present disclosure has been made to solve such a problem, and anobject is to accurately generate, with a simpler configuration,information in which eye gaze information and information regarding aperson are associated with each other.

One aspect of the present disclosure is an information processing methodin an information processing device, the information processing methodincluding: for each of one or more users, acquiring image data includingan eye of each of the users; detecting eye gaze information indicatingan eye gaze of each of the users based on information indicating the eyeof each of the users included in the image data; performing personalauthentication on each of the users based on information indicating theeye of each of the users included in the image data; acquiring personalinformation for identifying each of the users for which the personalauthentication has been performed; generating management information inwhich the personal information of the one or more users and the eye gazeinformation of the one or more users are associated with each other; andoutputting the management information.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view showing an example of an overall configuration of animage processing system according to a first embodiment of the presentdisclosure.

FIG. 2 is a block diagram showing an example of a detailed configurationof the image processing system according to the first embodiment.

FIG. 3 is a view showing an example of an eye region.

FIG. 4 is a view showing an example of an authentication informationtable.

FIG. 5 is a view showing an example of a user information table.

FIG. 6 is a flowchart showing an example of an operation of an imageprocessing device according to the first embodiment.

FIG. 7 is a view showing an example of a management information table.

FIG. 8 is a view showing another example of a management informationtable.

FIG. 9 is a flowchart showing an example of an operation of an imageprocessing device according to a fifth embodiment.

FIG. 10 is a flowchart showing an example of the operation of the imageprocessing device according to the fifth embodiment.

FIG. 11 is a view showing an example of a temporary managementinformation table.

FIG. 12 is a block diagram showing an example of a detailedconfiguration of the image processing system according to a sixthembodiment.

DESCRIPTION OF EMBODIMENTS Findings Underlying Present Disclosure

In the technique disclosed in Patent Literature 1 described above, inorder to generate a heat map indicative of an attention degree of thepurchaser relative to a product, a store is divided into a plurality ofareas, and information in which an attribute of the customer isassociated with a movement line (stopped-by area or the like) of thecustomer, information in which a product arranged in each area isassociated with a position to which an eye gaze of the customer isoriented, and the like are used. Wireless sensor cameras installed on aceiling and a wall surface of a store are used in order to acquireinformation regarding attributes and movement lines of customers. An eyegaze sensor attached to a product display shelf is used in order toacquire information indicating an eye gaze of a customer.

Therefore, the technique disclosed in Patent Literature 1 has a problemthat in order to generate information in which eye gaze information of acustomer and behavior information of the customer are associated witheach other, equipment used for acquiring the eye gaze information of thecustomer and the behavior information of the customer becomes large inscale. In the technique disclosed in Patent Literature 1, pieces ofinformation acquired at different timings in a plurality of pieces ofequipment are combined stepwise to obtain information in which eye gazeinformation and behavior information are associated with each other. Forthis reason, the processing of combining the information becomescomplicated, resulting in a problem that the accuracy of the temporalcorrespondence relationship between the eye gaze information and thebehavior information may decrease.

Therefore, as a result of conducting detailed studies on such a problem,the present inventor has obtained a finding that information in whicheye gaze information and information regarding a person are associatedwith each other can be accurately generated with a simpler configurationby using an image including an eye of the user not only for detection ofthe eye gaze information but also for personal authentication, and thepresent inventor has conceived of the following aspects.

An information processing method according to one aspect of the presentdisclosure is an information processing method in an informationprocessing device, the information processing method including: for eachof one or more users, acquiring image data including an eye of each ofthe users; detecting eye gaze information indicating an eye gaze of eachof the users based on information indicating the eye of each of theusers included in the image data; performing personal authentication oneach of the users based on information indicating the eye of each of theusers included in the image data; acquiring personal information foridentifying each of the users for which the personal authentication hasbeen performed; generating management information in which the personalinformation of the one or more users and the eye gaze information of theone or more users are associated with each other; and outputting themanagement information.

In the present configuration, for each of one or more users, based oninformation indicating an eye of each user included in image dataincluding the eye of each user, detection of eye gaze information andpersonal authentication are performed, and the personal information ofeach user is acquired. Then, the present configuration generates andoutputs management information in which the personal information of oneor more users thus acquired and the eye gaze information of one or moreusers are associated with each other.

Therefore, with the present configuration, the image data used forgenerating the management information in which the eye gaze informationand the personal information of each user are associated with each othercan be limited only to the image data including the eye of each user. Asa result, with the present configuration, the information in which theeye gaze information of each user is associated with the personalinformation of each user can be generated with a simpler configuration.

Furthermore, in the present configuration, the eye gaze information ofeach user and the image data used for personal authentication are thesame, it is possible to detect the eye gaze information and perform thepersonal authentication, based on the information indicating the eye ofeach user at the same time point. This makes it possible to acquire theeye gaze information and the personal information having no temporaldifference with respect to the user who has been subjected to thepersonal authentication, and to generate information in which the eyegaze information and the personal information are associated with eachother. Therefore, based on the information indicating the eye of eachuser at different time points from each other, the present configurationcan generate information in which the eye gaze information and thepersonal information of each user are associated with each other withhigher accuracy than in a case where the detection of the eye gazeinformation and the personal authentication are performed.

In the above aspect, the personal information may include one or moreattributes indicating a nature or a feature of each of the users. Inoutput of the management information, based on the managementinformation, eye gaze usage information in which the eye gazeinformation is classified for each of the one or more attributes may befurther generated, and the eye gaze usage information may be output.

According to the present configuration, further, the eye gaze usageinformation in which the eye gaze information is classified for each ofone or more attributes based on the management information is generatedand output. Therefore, the viewer of the eye gaze usage informationhaving been output can easily grasp the tendency of the eye gaze of theuser having the same one or more attributes.

In the above aspect, the one or more attributes may include one or moreof an age, a gender, a work place, and a job type.

According to the present configuration, the eye gaze usage informationin which the eye gaze information is classified by one or more of theage, the gender, the work place, and the job type is generated andoutput. Therefore, the viewer of the eye gaze usage information havingbeen output can easily grasp the tendency of the eye gaze of the userhaving the same one or more attributes of the age, the gender, the workplace, and the job type.

In the above aspect, the eye gaze information may include eye gazeposition information indicating a position to which an eye gaze of eachof the users is oriented, and the eye gaze usage information may be aheat map representing a relationship between a position indicated by theeye gaze position information and a frequency at which the eye gaze ofthe user is oriented to a position indicated by the eye gaze positioninformation.

According to the present configuration, the heat map representing therelationship between the position indicated by the eye gaze positioninformation and the frequency at which the eye gaze of the user isoriented to the position indicated by the eye gaze position informationis output as the eye gaze usage information. Therefore, the viewer ofthe heat map having been output can easily grasp which position the eyegaze of the user having the same attribute is frequently oriented to.

In the above aspect, the eye gaze information may include eye gazeposition information indicating a position to which the eye gaze of eachof the users is oriented, and the eye gaze usage information may be agaze plot representing a relationship among the position indicated bythe eye gaze position information, a number of times the eye gaze of theuser is oriented to the position indicated by the eye gaze positioninformation, and a movement route of the eye gaze of the user to theposition indicated by the eye gaze position information.

According to the present configuration, the gaze plot representing therelationship among the position indicated by the eye gaze positioninformation, the number of times the eye gaze of the user is oriented tothe position indicated by the eye gaze position information, and themovement route of the eye gaze of the user to the position indicated bythe eye gaze position information is output as the eye gaze usageinformation. Therefore, the viewer of the gaze plot having been outputcan easily grasp which position on which movement route the eye gaze ofthe user having the same attribute is oriented to many times.

In the above aspect, in detection of the eye gaze information,information indicating the eye of each of the users and informationindicating the orientation of the face of each of the users may bedetected from the image data, and the eye gaze information may bedetected based on the detected information indicating the eye of each ofthe users and the detected information indicating the orientation of theface of each of the users.

According to the present configuration, the information indicating theeye of each user and the information indicating the orientation of theface of each user are detected from the image data including the eye ofeach user, and the eye gaze information is detected based on thedetected information. Thus, the present configuration can accuratelydetect the eye gaze of each user from the information indicating the eyeand the orientation of the face obtained from the image data.

In the above aspect, in personal authentication of each of the users,iris information indicating an iris of the eye of each of the users maybe detected from the image data, and each of the users may be subjectedto the personal authentication based on the detected iris information.

According to the present configuration, the iris information indicatingthe iris of the eye of each user is detected from the image dataincluding the eye of each user, and each user is subjected to thepersonal authentication based on the detected iris information. Thus, inthe present configuration, it is possible to accurately perform personalauthentication of each user based on the iris unique to each user.

In the above aspect, the one or more users may be participants in anexhibition, the one or more attributes may include a work place of theparticipants, the eye gaze information may include exhibit informationindicating an exhibit of the exhibition existing at a position to whichan eye gaze of each of the users is oriented, and the eye gaze usageinformation may be a heat map representing a relationship between anexhibit of the exhibition indicated by the exhibit information and afrequency at which the eye gaze of the user is oriented to the exhibitof the exhibition.

In the present configuration, one or more users are participants of anexhibition, and the attribute of each user includes the work place ofthe participant. In addition, a heat map representing the relationshipbetween an exhibit of the exhibition indicated by the exhibitinformation and the frequency at which the eye gaze of the user isoriented to the exhibit of the exhibition is output as the eye gazeusage information. For this reason, the viewer of the heat map havingbeen output can easily grasp, for example, in the exhibition, an eyegaze of a participant of which work place is highly frequently orientedto which exhibit.

In the above aspect, the one or more users may be workers at amanufacturing site, the one or more attributes may include workproficiency of the workers, the eye gaze information may include worktarget information indicating a work target present at a position towhich an eye gaze of each of the users is oriented, and the eye gazeusage information may be a heat map representing a relationship betweenthe work target indicated by the work target information and a frequencyat which the eye gaze of the user is oriented to the work target.

In the present configuration, the one or more users are workers at amanufacturing site, and the attribute of each user includes the workproficiency of the worker. Furthermore, a heat map representing arelationship between the work target indicated by the work targetinformation and the frequency at which the eye gaze of the user isoriented to the work target is output as the eye gaze usage information.Therefore, the viewer of the heat map having been output can easilygrasp, for example, at the manufacturing site, which work target an eyegaze of a highly proficient worker is frequently oriented to.

In the above aspect, the image data may be captured by an infrared lightcamera.

In the image data captured by the infrared light camera, luminancechange of the outer edge of each of the pupil and the iris tends toappear clearly. Furthermore, in the present configuration, each user issubjected to personal authentication based on information indicating theeye of each user included in the image data captured by the infraredlight camera. Therefore, according to the present configuration, theiris information indicating the iris of the eye of each user can beaccurately detected from the image data as the information indicatingthe eye of each user used for personal authentication. As a result, itis possible for the present configuration to accurately perform personalauthentication of each user.

The present disclosure can also be implemented as a control program forcausing a computer to execute each characteristic configuration includedin such an information processing method, or an information processingdevice operated by this control program. Furthermore, it goes withoutsaying that such a control program can be distributed via acomputer-readable non-transitory recording medium such as a CD-ROM or acommunication network such as the internet.

Note that each of the embodiments described below shows a specificexample of the present disclosure. Numerical values, shapes, constituentelements, steps, orders of steps, and the like shown in the followingembodiments are merely examples, and are not intended to limit thepresent disclosure. Among the constituent elements in the followingembodiments, constituent elements that are not described in independentclaims indicating the highest concept are described as discretionaryconstituent elements. In addition, in all the embodiments, each of thecontents can be combined.

First Embodiment

FIG. 1 is a view showing an example of an overall configuration of animage processing system 1 according to the first embodiment of thepresent disclosure. The image processing system 1 is a system thatcaptures a person 400 and detects eye gaze information indicating an eyegaze of the person 400 from the obtained image data of the person 400.In the example of FIG. 1, the image processing system 1 specifies whichobject 301 the person 400 gazes at among a plurality of objects 301displayed on a display device 300. However, this is an example, and theimage processing system 1 may specify not only the object 301 displayedon the display screen of the display device 300 but also the object 301gazed by the person 400 in the real space.

In the example of FIG. 1, the image processing system 1 is applied to adigital signage system. Therefore, the object 301 displayed on thedisplay device 300 is an image of signage such as an advertisement.Furthermore, the image processing system 1 generates and outputsinformation, obtained based on the image data of the person 400, inwhich information indicating the eye gaze of the person 400 isassociated with the personal information of the person 400.

The image processing system 1 includes an image processing device 100(an example of an information processing device), a camera 200, and adisplay device 300. The image processing device 100 is connected to thecamera 200 and the display device 300 via a predetermined communicationpath. The predetermined communication path is, for example, a wiredcommunication path such as a wired LAN, or a wireless communication pathsuch as a wireless LAN and Bluetooth (registered trademark). The imageprocessing device 100 includes, for example, a computer installed aroundthe display device 300. However, this is an example, and the imageprocessing device 100 may include a cloud server. In this case, theimage processing device 100 is connected to the camera 200 and thedisplay device 300 via the Internet. The image processing device 100detects eye gaze information of the person 400 from the image data ofthe person 400 captured by the camera 200, and outputs the eye gazeinformation to the display device 300. Furthermore, the image processingdevice 100 may be incorporated as hardware in the camera 200 or thedisplay device 300. Furthermore, the camera 200 or the display device300 may include a processor, and the image processing device 100 may beincorporated as software.

By capturing an image of an environment around the display device 300 ata predetermined frame rate, for example, the camera 200 acquires imagedata of the person 400 positioned around the display device 300. Thecamera 200 sequentially outputs the acquired image data to the imageprocessing device 100 at a predetermined frame rate. The camera 200 maybe a visible light camera or may be an infrared light camera.

The display device 300 includes a display device such as a liquidcrystal panel or an organic EL panel. In the example of FIG. 1, thedisplay device 300 is a signage display. Note that in the example ofFIG. 1, the image processing system 1 is described to include thedisplay device 300, but this is an example, and another piece ofequipment may be adopted instead of the display device 300. For example,if the image processing system 1 is used as a user interface thatreceives an input to equipment by an eye gaze, the image processingsystem 1 may adopt home appliances such as a refrigerator, a televisionset, and a washing machine instead of the display device 300, forexample. For example, if the image processing system 1 is mounted on avehicle, a vehicle such as an automobile may be adopted instead of thedisplay device 300. Furthermore, a storage device such as a hard diskdrive or a solid state drive may be adopted instead of the displaydevice 300.

FIG. 2 is a block diagram showing an example of a detailed configurationof the image processing system 1 according to the first embodiment. Theimage processing device 100 includes a processor 120 and a memory 140.

The processor 120 is an electric circuit such as a CPU or an FPGA. Theprocessor 120 includes an image acquisition unit 121, an eye detectionunit 122, an iris authentication unit 123 (an example of theauthentication unit), a facial feature detection unit 124, an eye gazedetection unit 125, a management information generation unit 126 (a partof the personal information acquisition unit), and an output unit 127.Note that each block included in the processor 120 may be implemented bythe processor 120 executing a control program for causing a computer tofunction as an image processing device, or may be configured by adedicated electric circuit.

The image acquisition unit 121 acquires image data captured by thecamera 200. Here, the acquired image data includes the face of theperson 400 (an example of the user) around the display device 300. Notethat the image data acquired by the image acquisition unit 121 may be,for example, image data posted on a website or may be image data storedin an external storage device.

The eye detection unit 122 detects an eye region including the eye ofthe person 400 from the image data acquired by the image acquisitionunit 121. Specifically, the eye detection unit 122 is only required todetect the eye region using a classifier created in advance fordetecting the eye region. The classifier used here is a Haar-likecascade classifier created in advance for detecting the eye region in anopen-source image processing library, for example.

The eye region is a rectangular region having a size in which apredetermined margin is added to the size of the eye. However, this isan example, and the shape of the eye region may be, for example, atriangle, a pentagon, a hexagon, an octagon, or the like other than arectangle. Note that the position at which the boundary of the eyeregion is set with respect to the eye depends on the performance of theclassifier.

FIG. 3 is a view showing an example of an eye region 50. In the presentembodiment, the eye refers to a region including the white of the eyeand a colored part such as the iris that are surrounded by a boundary 53of the upper eyelid and a boundary 54 of the lower eyelid as shown inFIG. 3. As shown in FIG. 3, the colored part includes a pupil 55 and adonut-like iris 56 surrounding the pupil 55. In the present embodiment,for convenience of description, the right eye refers to the eye on theright side when the person 400 is viewed from the front, and the lefteye refers to the eye on the left side when the person 400 is viewedfrom the front. FIG. 3 shows an example in which the eye detection unit122 detects the eye region 50 including the right eye and the eye region50 including the left eye. However, this is an example, and the eye onthe right side as viewed from the person 400 may be the right eye andthe eye on the left side as viewed from the person 400 may be the lefteye. In the present embodiment, the direction on the right side of thepaper surface is defined as the right side, and the direction on theleft side of the paper surface is defined as the left side.

The iris authentication unit 123 detects iris information indicating theiris 56 of the eye of the person 400 in the eye region 50 detected bythe eye detection unit 122, and performs personal authentication of theperson 400 using the detected iris information and an authenticationinformation storage unit 141.

The iris information includes, for example, coordinate data indicatingthe outer edge of the iris 56 or information indicating a length (e.g.,a pixel) such as a radius or a diameter of the outer edge of the iris56, and coordinate data of the center of the iris 56. Here, thecoordinate data refers to two-dimensional coordinate data in the imagedata acquired by the image acquisition unit 121. The iris informationincludes iris data obtained by coding an image of the iris 56 with apredetermined algorithm such as a Daugman algorithm, for example.Daugman algorithm is disclosed in the document “High Confidence VisualRecognition of Persons by a Test of Statistical Independence: John G.Daugman (1993)”. Note that the iris data is not limited thereto, and maybe image data (binary data) in which an image of the iris 56 isrepresented in a predetermined file format (e.g., PNG).

If an infrared light camera is adopted as the camera 200, the luminancechanges between the pupil 55 and the iris 56 appears clearly Therefore,if an infrared light camera is adopted as the camera 200, the irisauthentication unit 123 may further detect, as the iris information,coordinate data indicating the outer edge of the pupil 55, for example,or information indicating a length (e.g., a pixel) such as a radius or adiameter of the outer edge of the pupil 55, and coordinate data of thecenter of the pupil 55. On the other hand, if a visible light camera isadopted as the camera 200, there is a case where a luminance changebetween the pupil 55 and the iris 56 does not appear clearly, and hence,it is difficult to distinguish between the pupil 55 and the iris 56.Therefore, if a visible light camera is adopted as the camera 200, theiris authentication unit 123 may not detect the coordinate data andinformation regarding the pupil 55 described above. Detail of thepersonal authentication of the person 400 using the iris information andthe authentication information storage unit 141 will be described later.

The facial feature detection unit 124 detects a facial feature point ofthe person 400 from the image data acquired by the image acquisitionunit 121. The facial feature point is one or a plurality of points atcharacteristic positions in each of a plurality of parts constitutingthe face such as the outer corner of the eye, the inner corner of theeye, the contour of the face, the ridge of the nose, the corner of themouth, and the eyebrow, for example.

Specifically, the facial feature detection unit 124 first detects a faceregion indicating the face of the person 400 from the image dataacquired by the image acquisition unit 121L For example, the facialfeature detection unit 124 is only required to detect the face regionusing a classifier created in advance for detecting the face region. Theclassifier used here is a Haar-like cascade classifier created inadvance for detecting the face region in an open-source image processinglibrary, for example. The face region is a rectangular region having asize enough to include the entire face, for example. However, this is anexample, and the shape of the face region may be, for example, atriangle, a pentagon, a hexagon, an octagon, or the like other than arectangle. Note that the facial feature detection unit 124 may detectthe face region by pattern matching.

Next, the facial feature detection unit 124 detects a facial featurepoint from the detected face region. The feature point is also called alandmark. The facial feature detection unit 124 is only required todetect a facial feature point by executing landmark detection processingusing a model file of a framework of machine learning, for example.

The eye gaze detection unit 125 detects information indicating the eyegaze (hereinafter, eye gaze information) of the person 400 based on thefacial feature point detected by the facial feature detection unit 124and the information indicating the eye of the person 400 included in theeye region 50 detected by the eye detection unit 122.

Specifically, by performing known face orientation detection processing,the eye gaze detection unit 125 detects face orientation informationindicating the orientation of the face of the person 400 from thearrangement pattern of the facial feature point detected by the facialfeature detection unit 124. The face orientation information includes anangle indicating the front direction of the face with respect to theoptical axis of the camera 200, for example.

Next, by performing known eye gaze detection processing for detecting aneye gaze by a three-dimensional eyeball model, the eye gaze detectionunit 125 detects the eye gaze information based on the above-describeddetected face orientation information and the information indicating theeye of the person 400 included in the eye region 50 detected by the eyedetection unit 122. The information indicating the eye includes, forexample, the positions of the colored part, the inner corner of the eye,the outer corner of the eye, and the center of gravity of the eye.Furthermore, the information indicating the eye includes, for example,iris information detected from the eye region 50 by the irisauthentication unit 123. The eye gaze information includes capturingdate and time of the image data used to detect the eye gaze informationand coordinate data of an eye gaze point on a predetermined target plane(e.g., the display device 300). The eye gaze point is a position towhich the eye gaze of the person 400 is oriented, and is, for example, aposition where a target plane and a vector indicating the eye gazeintersect. Note that the eye gaze information may include a vectorindicating the direction of the eye gaze of the person 400 instead ofthe coordinate data of the eye gaze point or in addition to thecoordinate data of the eye gaze point. The vector is only required to beexpressed by, for example, an angle of a horizontal component withrespect to a reference direction such as an optical axis direction ofthe camera 200 and an angle in a vertical direction with respect to thereference direction.

The management information generation unit 126 acquires, from a userinformation storage unit 142, personal information for identifying theuser who has been subjected to the personal authentication each time theuser of the image processing system 1 is captured by the camera 200 andthe user is subjected to the personal authentication by the irisauthentication unit 123. Furthermore, when the eye gaze detection unit125 detects the eye gaze information from the image data obtained bycapturing the user who has been subjected to the personalauthentication, the management information generation unit 126 generatesinformation (hereinafter, eye gaze management information) in which thedetected eye gaze information is associated with the acquired personalinformation. Details of the acquisition of the personal informationusing the user information storage unit 142 and the generation of theeye gaze management information will be described later.

The output unit 127 outputs, to the display device 300, the eye gazeinformation detected by the eye gaze detection unit 125. The output unit127 may acquire information of the object 301 displayed on the displaydevice 300, specify the object 301 (hereinafter, gaze object) at whichthe person 400 gazes from the acquired information and the coordinatedata of the eye gaze point, and output the specification result to thedisplay device 300.

In addition, the output unit 127 stores (an example of outputting) theeye gaze management information for one or more users generated by themanagement information generation unit 126 in a memory (not illustrated)included in the processor 120 or a storage device (not illustrated) suchas a hard disk drive or a solid state drive included in the imageprocessing device 100. Note that the output unit 127 may output, to thedisplay device 300, the eye gaze management information for one or moreusers generated by the management information generation unit 126.

The memory 140 is a storage device such as a hard disk drive or a solidstate drive. The memory 140 includes the authentication informationstorage unit 141 and the user information storage unit 142.

The authentication information storage unit 141 stores an authenticationinformation table in advance. The authentication information table is atable in which the iris authentication unit 123 stores authenticationinformation used for personal authentication of the user of the imageprocessing system 1.

FIG. 4 is a view showing an example of an authentication informationtable T1. Specifically, as shown in FIG. 4, the authenticationinformation stored in the authentication information table T1 includes“user ID”, “iris ID”, “iris data”, “pupil diameter size”, and “irisdiameter size”. The “user ID” is an identifier uniquely allocated to theuser of the image processing system 1. The “iris ID” is an identifieruniquely allocated to the “iris data”. The “iris data” is data obtainedby coding an image of the iris 56 of the user of the image processingsystem 1 with a predetermined algorithm such as a Daugman algorithm.

The “pupil diameter size” is the diameter of an outer edge of the pupil55 of the user of the image processing system 1. The “iris diametersize” is the diameter of an outer edge of the iris 56 of the user of theimage processing system 1. Note that the authentication informationtable T1 is only required to store at least the “user ID”, the “irisID”, and the “iris data”, and may not store one or more of the “pupildiameter size” and the “iris diameter size”.

The user information storage unit 142 stores a user information table inadvance. The user information table is a table that stores personalinformation of the user of the image processing system 1.

FIG. 5 is a view showing an example of a user information table T2.Specifically, as shown in FIG. 5, the personal information stored in theuser information table T2 includes “user ID”, “privacy information”, and“attribute information”. The “user ID” is an identifier uniquelyallocated to the user of the image processing system 1. The “privacyinformation” is information regarding privacy that can uniquely identifythe user of the image processing system 1. In the example of FIG. 5, the“privacy information” includes “name”, “address”, “telephone number”,and “mail address”. The “name”, the “address”, the “telephone number”,and the “mail address” are a name, an address, a telephone number, and amail address of the user of the image processing system 1, respectively.The “attribute information” is information indicating one or moreattributes indicating the nature or feature of the user of the imageprocessing system 1. In the example of FIG. 5, the “attributeinformation” includes “age”, “gender”, “work place”, and “job type”. The“age,” the “gender,” the “work place,” and the “job type” are the age,the gender, the work place, and the job type of the user of the imageprocessing system 1, respectively. The “attribute information” is notlimited thereto, and is only required to include one or more of “age”,“gender”, “work place”, and “job type”.

Since the camera 200 has been described with reference to FIG. 1, thedescription thereof is omitted here.

The display device 300 displays a marker indicating the eye gazeinformation output from the output unit 127. The display device 300 maydisplay a marker indicating the object 301 gazed by the person 400output from the output unit 127. For example, it is assumed thatcoordinate data of the eye gaze point is output to the display device300 as eye gaze information. In this case, the display device 300performs processing of displaying, at a position corresponding to thecoordinate data, a marker indicating the eye gaze position superimposedon the screen being displayed. For example, it is assumed that aspecification result of the eye gaze object is output to the displaydevice 300. In this case, the display device 300 may perform processingof displaying a marker indicating the eye gaze object superimposed onthe screen being displayed. Furthermore, the display device 300 maydisplay the eye gaze management information regarding one or more usersoutput from the output unit 127.

Note that, in a case where the image processing system 1 includes a homeappliance instead of the display device 300, the home appliance receivesan input of the person 400 from the eye gaze information. Furthermore,in a case where the image processing system 1 includes a storage deviceinstead of the display device 300, the storage device stores the eyegaze information. In this case, the storage device may store the eyegaze information in association with a time stamp.

Next, the operation of the image processing device 100 will bedescribed. FIG. 6 is a flowchart showing an example of the operation ofthe image processing device 100 according to the first embodiment. Theoperation of the image processing device 100 shown in FIG. 6 is startedperiodically (e.g., every second). When the operation of the imageprocessing device 100 is started and the image acquisition unit 121acquires image data of the face of the person 400 from the camera 200(step S1), the eye detection unit 122 detects the eye region 50 from theimage data by inputting the image data acquired in step S1 to aclassifier for detecting the eye region 50 (step S2).

Next, the iris authentication unit 123 detects iris informationindicating the iris 56 of the eye of the person 400 in the eye region 50detected in step S2, and performs personal authentication of the person400 using the detected iris information and the authenticationinformation storage unit 141 (step S3).

Specifically, in step S3, the iris authentication unit 123 refers,record by record, to the authentication information table T1 (FIG. 4)stored in the authentication information storage unit 141. Next, theiris authentication unit 123 calculates a ratio (hereinafter, the firstratio) between the length of the diameter of the outer edge of the pupil55 included in the detected iris information and the length of thediameter of the outer edge of the iris 56 included in the detected irisinformation. Furthermore, the iris authentication unit 123 calculates aratio (hereinafter, the second ratio) between the “pupil diameter size”included in the referred record and the “iris diameter size” included inthe referred record.

Then, the iris authentication unit 123 determines whether or not thedifference between the first ratio and the second ratio is equal to orless than a predetermined first threshold value. When it is determinedthat the difference between the first ratio and the second ratio isequal to or less than the first threshold value, the iris authenticationunit 123 determines whether or not the similarity between the iris dataincluded in the detected iris information and the “iris data” of thereferred record is equal to or greater than a predetermined secondthreshold value. When it is determined that the similarity is equal toor greater than the second threshold value, the iris authentication unit123 performs personal authentication that the person 400 is a user ofthe image processing system 1 identified by the “user ID” included inthe referred record. Then, as the user ID of the user who has beensubjected to the personal authentication, the iris authentication unit123 outputs the “user ID” of the referred record.

Next, the management information generation unit 126 acquires thepersonal information of the person 400 who has been subjected to thepersonal authentication in step S3 (step S4). Specifically, in step S4,in the user information table T2 (FIG. 5) stored in advance in the userinformation storage unit 142, the management information generation unit126 acquires, as the personal information of the person 400, the recordincluding the “user ID” matching the user ID of the user who has beensubjected to the personal authentication, output by the irisauthentication unit 123 in step S3. In the example of FIG. 5, when theuser ID of the person 400 who has been subjected to the personalauthentication is “U001”, the management information generation unit 126acquires, as the personal information of the person 400, a record in afirst line including the user ID “U001” which matches the user ID, the“privacy information” in which the “name” is “aYAMA bTA”, and the“attribute information” in which the “age” is “45”.

Next, the facial feature detection unit 124 detects a facial featurepoint of the person 400 from the image data acquired by the imageacquisition unit 121 in step S1 (step S5). Next, the eye gaze detectionunit 125 detects eye gaze information based on the facial feature pointdetected in step S5 and the information indicating the eye of the person400 included in the eye region 50 detected in step S2 (step S6).

Specifically, in step S6, the eye gaze detection unit 125 detects faceorientation information indicating the orientation of the face of theperson 400 from the arrangement pattern of the facial feature pointdetected by the facial feature detection unit 124 performing known faceorientation detection processing in step S5. Next, by performing knowneye gaze detection processing for detecting an eye gaze by athree-dimensional eyeball model, the eye gaze detection unit 125 detectsthe eye gaze information based on the detected face orientationinformation and the information indicating the eye of the person 400included in the eye region 50 detected in step S2. In the presentembodiment, the eye gaze information detected in step S6 is assumed toinclude coordinate data indicating the position of the eye gaze point onthe display device 300 and information for identifying the object 301displayed at the position of the eye gaze point on the display device300.

Next, the management information generation unit 126 generates eye gazemanagement information in which the eye gaze information detected instep S6 is associated with the personal information acquired in step S5(step S7). The output unit 127 stores the eye gaze managementinformation generated in step S7 into a management information table (anexample of the management information) (step Sg). The managementinformation table is a table that stores eye gaze management informationregarding one or more persons 400 generated by the managementinformation generation unit 126. The management information table isstored in a memory (not illustrated) included in the processor 120 or astorage device (not illustrated) such as a hard disk drive or a solidstate drive included in the image processing device 100.

FIG. 7 is a view showing an example of a management information tableT3. For example, in step S7, as shown in FIG. 7, the managementinformation generation unit 126 generates eye gaze managementinformation in which “image capturing date and time”, “eye gaze positionX coordinate”, “eye gaze position Y coordinate”, and “gazed object ID”included in the eye gaze information detected in step S6 are associatedwith “user ID”, “age”, “gender”, “work place”, and “job type” includedin the personal information acquired in step S5. The output unit 127stores, in the management information table 13, the eye gaze managementinformation generated by the management information generation unit 126.

The “image capturing date and time” is the acquisition date and time ofthe image data used to detect the eye gaze information, i.e., the dateand time when the image data is acquired in step S1. The “eye gazeposition X coordinate” is a horizontal component of the coordinate dataindicating the position of the eye gaze point on the display device 300,and the “eye gaze position Y coordinate” is a vertical component of thecoordinate data indicating the position of the eye gaze point. The“gazed object ID” is information for identifying the object 301displayed at the position of the eye gaze point on the display device300. The “age”, the “gender”, the “work place”, and the “job type” areinformation stored in advance as the attribute information in the userinformation table T2 (FIG. 5). Thus, in the present specific example,the eye gaze management information in which the “privacy information”included in the personal information is not associated with the eye gazeinformation but the “attribute information” included in the personalinformation is associated with the eye gaze information is generated.This makes it possible to generate the eye gaze management informationwith contents in which privacy is protected.

In the example of FIG. 7, in step S1, the image data of the face of theuser whose “user ID” is “U001” when the date and time is “2019/5/1713:33:13” is acquired. The eye gaze management information in which theeye gaze information whose “eye gaze position X coordinate” detectedfrom the image data is “1080” is associated with the personalinformation whose “user ID” is “U001” is generated and stored in themanagement information table T3. In this manner, in the example of FIG.7, the management information table T3 stores the eye gaze managementinformation for a total of 11 persons 400 including the same person 400.

The eye gaze management information generated in step S7 is not limitedto the above. FIG. 8 is a view showing another example of the managementinformation table T3. For example, as shown in FIG. 8, the managementinformation generation unit 126 may generate the eye gaze managementinformation in which “image capturing date and time”, “eye gaze positionX coordinate”, “eye gaze position Y coordinate”, and “gazed object ID”included in the eye gaze information detected in step S6 are associatedwith information (“user ID”) in which the “privacy information” (FIG. 5)and the “attribute information” (FIG. 5) are removed from the personalinformation acquired in step S5. Alternatively, step S4 may be omitted,and in step S7, the “user ID” of the user who has been subjected to thepersonal authentication in step S3 may be associated, as the personalinformation, with the eye gaze information detected in step S6.

In this manner, by removing the “privacy information” (FIG. 5) and the“attribute information” (FIG. 5) from the personal information to beassociated with the eye gaze information, the time required forgeneration of eye gaze management information may be further shortened.In addition, after the generation of the eye gaze managementinformation, step S4 may be performed using the “user ID” included inthe eye gaze management information at an arbitrary timing. Then, thepersonal information acquired in step S4 may be added to the eye gazemanagement information including the “user ID” used in step S4. In thismanner, the details of the personal information of the authenticateduser may be added as the eye gaze management information afterwards.

Thus, when the information indicating the vector indicating thedirection of the eye gaze of the person 400 is included in the eye gazeinformation as described above, the management information generationunit 126 may generate the eye gaze management information in which theinformation indicating the vector is associated with the personalinformation. In addition, the management information generation unit 126may include an identifier for uniquely specifying the eye gazemanagement information in the generated eye gaze management information.

As described above, according to the present embodiment, for each of theone or more users of the image processing system 1, the detection of theeye gaze information and the personal authentication are performed basedon the information indicating the eye of each user included in the imagedata including the eye of each user, and the personal information ofeach user is acquired. In the present embodiment, the eye gazemanagement information in which the thus acquired personal informationand the eye gaze information are associated with each other isgenerated. In this manner, the result of generation of the eye gazemanagement information for one or more users is stored in the managementinformation table T3.

Therefore, in the present embodiment, the image data used for generatingthe eye gaze management information in which the eye gaze informationand the personal information of each user are associated with each othercan be limited only to the image data including the eye of each user.Thus, in the present embodiment, the information in which the eye gazeinformation of each user is associated with the personal information ofeach user can be generated with a simpler configuration.

Furthermore, in the present embodiment, since the eye gaze informationof each user and the image data used to acquire the personal informationare the same, it is possible to detect the eye gaze information andperform the personal authentication based on the information indicatingthe eye of each user at the same time point. This enables the presentconfiguration to acquire eye gaze information and personal informationhaving no temporal difference regarding the user having been subjectedto the personal authentication, and to generate the eye gaze managementinformation in which the eye gaze information and the personalinformation are associated with each other.

Therefore, based on the information indicating the eye of each user atdifferent time points from each other, the present configuration cangenerate information in which the eye gaze information and the personalinformation of each user are associated with each other with higheraccuracy than in a case where the detection of the eye gaze informationand the personal authentication are performed.

Second Embodiment

In the second embodiment, the output unit 127 further generates eye gazeusage information in which the eye gaze information is classified foreach of one or more attributes based on the eye gaze managementinformation for one or more users generated by the managementinformation generation unit 126, and outputs the eye gaze usageinformation.

For example, as shown in FIG. 7, it is assumed that the managementinformation table T3 stores 11 pieces of eye gaze management informationregarding users with the user IDs “U001”, “U002”, and “U003”. In thiscase, for example, the output unit 127 classifies the 11 pieces of eyegaze management information by “gender”, and generates, as the eye gazeusage information, six pieces of eye gaze management information withthe “user ID” of “U001” and “U003”, in which “gender” is “male”. Then,the output unit 127 outputs the six pieces of eye gaze managementinformation to the display device 300 as the eye gaze usage informationtogether with the information indicating that the “gender” is “male”.

Similarly, the output unit 127 generates, as the eye gaze usageinformation, five pieces of eye gaze management information with the“user ID” of “U002” in which the “gender” is “female”, and displays, asthe eye gaze usage information, the five pieces of eye gaze managementinformation together with the information indicating that the “gender”is “female” in this case, the output unit 127 may display theinformation indicating that the “gender” is “female” in a colordifferent from that of the information indicating that the “gender” is“male”, whereby make the display mode of the eye gaze usage informationdifferent according to the attribute corresponding to the eye gaze usageinformation of the display target. According to the present embodiment,the viewer of the eye gaze usage information can easily grasp thetendency of the eye gaze of the user having the same one or moreattributes.

Third Embodiment

In the third embodiment, in a case where, in the second embodiment, forexample, as shown in FIG. 7, the coordinate data of the eye gaze pointis included in the eye gaze information included in the eye gazemanagement information, the output unit 127 outputs, to the displaydevice 300 as the eye gaze usage information, a heat map representingthe relationship between the eye gaze point indicated by the coordinatedata included in the eye gaze information and the frequency at which theeye gaze of the user is oriented to the eye gaze point.

Hereinafter, a method in which the output unit 127 outputs theabove-described beat map to the display device 300 as the eye gaze usageinformation will be described with reference to FIG. 7. First, theoutput unit 127 classifies the 11 pieces of eye gaze managementinformation shown in FIG. 7 by “gender”, generates six pieces of eyegaze management information with the “user ID” of “U001” and “U003” inwhich “gender” is “male” as the first eye gaze usage information, andgenerates five pieces of eye gaze management information with the “userID” of “U002” in which “gender” is “female” as the second eye gaze usageinformation.

Next, for each of the first eye gaze usage information and the secondeye gaze usage information, the output unit 127 refers to the eye gazeinformation in each piece of eye gaze management information included ineach piece of eye gaze usage information, and calculates the frequencyat which the eye gaze of the user is oriented to the eye gaze point(hereinafter, target eye gaze point) indicated by the coordinate dataincluded in the referred eye gaze information.

Specifically, as the frequency at which the eye gaze of the user isoriented to the target eye gaze point, the output unit 127 calculatesthe frequency at which the eye gaze of the user is oriented to theobject 301 (hereinafter, the target object) including the target eyegaze point.

For example, the first eye gaze usage information includes six pieces ofeye gaze management information, where there are four pieces of eye gazemanagement information with the “gazed object ID” of “C001”, one pieceof eye gaze management information with the “gazed object ID” of “C002”,and one piece of eye gaze management information with the “gazed objectID” of “C003”. In this case, the output unit 127 calculates, as “ 4/6”,the frequency at which the eye gaze of the user is oriented to thetarget object having the “gazed object ID” of “C001”. Then, the outputunit 127 sets the calculated frequency “ 4/6” as a frequency at whichthe eye gaze of the user is oriented to each of the four target eye gazepoints with the “image capturing date and time” of “2019/5/17 13:33:13”to “2019/5/17 13:33:16” included in the target object with the “gazedobject ID” of “C001”.

Similarly, the output unit 127 calculates, as “⅙”, the frequency atwhich the eye gaze of the user is oriented to one target eye gaze pointwith the “image capturing date and time” of “2019/5/17 13:33:20”included in the target object with the “gazed object ID” of “C002”. Inaddition, the output unit 127 calculates, as “⅙”, the frequency at whichthe eye gaze of the user is oriented to one target eye gaze point withthe “image capturing date and time” of “2019/5/17 13:33:22” included inthe target object with the “gazed object ID” of “C003”.

Similarly, for the second eye gaze usage information, the output unit127 calculates, as “⅗”, the frequency at which the eye gaze of the useris oriented to the three target eye gaze points with the “imagecapturing date and time” of “2019/5/17 13:33:17” to “2019/5/17 13:33:19”included in the target object with the “gazed object ID” of “C004”. Inaddition, the output unit 127 calculates, as “⅕”, the frequency at whichthe eye gaze of the user is oriented to one target eye gaze point withthe “image capturing date and time” of “2019/5/17 13:33:21” included inthe target object with the “gazed object ID” of “C002”. In addition, theoutput unit 127 calculates, as “⅕”, the frequency at which the eye gazeof the user is oriented to one target eye gaze point with the “imagecapturing date and time” of “2019/5/17 13:33:23” included in the targetobject with the “gazed object ID” of “C003”.

Next, the output unit 127 displays, on the display device 300, eachtarget eye gaze point included in the first eye gaze usage informationin a more highlighted manner as the frequency at which the eye gaze ofthe user is oriented to each target eye gaze point is higher.

For example, the output unit 127 displays four target eye gaze pointswhose “image capturing date and time” are “2019/5/17 13:33:13” to“2019/5/17 13:33:16” and whose frequency is “ 4/6” in a more highlightedmanner than one target eye gaze point whose “image capturing date andtime” is “2019/5/17 13:33:20” and whose frequency is “⅙” and one targeteye gaze point whose “image capturing date and time” is “2019/5/1713:33:22” and whose frequency is “⅙”.

Similarly, the output unit 127 displays, on the display device 300, eachtarget eye gaze point included in the second eye gaze usage informationin a more highlighted manner as the frequency at which the eye gaze ofthe user is oriented to each target eye gaze point is higher. Forexample, the output unit 127 displays three target eye gaze points whose“image capturing date and time” are “2019/5/17 13:33:17” to “2019/5/1713:33:19” and whose frequency is “⅗” in a more highlighted manner thanone target eye gaze point whose “image capturing date and time” is“2019/5/17 13:33:21” and whose frequency is “⅕” and one target eye gazepoint whose “image capturing date and time” is “2019/5/17 13:33:23” andwhose frequency is “⅕”.

According to the present configuration, the viewer of the display device300 can easily grasp which position the frequency at which the eye gazeof the user having the same attribute is oriented to is high.

Fourth Embodiment

In the fourth embodiment, in a case where, in the second embodiment, forexample, as shown in FIG. 7, the coordinate data of the eye gaze pointis included in the eye gaze information included in the eye gazemanagement information, the output unit 127 outputs, to the displaydevice 300 as the eye gaze usage information, a gaze plot representingthe relationship among the eye gaze point indicated by the coordinatedata included in the eye gaze information, the number of times at whichthe eye gaze of the user is oriented to the eye gaze point, and themovement route of the eye gaze of the user to the eye gaze point.

Hereinafter, a method in which the output unit 127 outputs theabove-described gaze plot to the display device 300 as eye gaze usageinformation will be described with reference to FIG. 7. First, similarlyto the third embodiment, the output unit 127 classifies the 11 pieces ofeye gaze management information shown in FIG. 7 by “gender”, generatessix pieces of eye gaze management information in which “gender” is“male” as the first eye gaze usage information, and generates fivepieces of eye gaze management information in which “gender” is “female”as the second eye gaze usage information.

Next, for each of the first eye gaze usage information and the secondeye gaze usage information, the output unit 127 refers to the eye gazeinformation in each piece of eye gaze management information included ineach piece of eye gaze usage information, and calculates the number oftimes the eye gaze of the user is oriented to the target eye gaze pointindicated by the coordinate data included in the referred eye gazeinformation.

Specifically, the output unit 127 calculates, as the number of times theeye gaze of the user is oriented to the target eye gaze point, thenumber of times the eye gaze of the user is oriented to the targetobject including the target eye gaze point.

For example, the first eye gaze usage information includes six pieces ofeye gaze management information, where there are four pieces of eye gazemanagement information with the “gazed object ID” of “C001”, one pieceof eye gaze management information with the “gazed object ID” of “C002”,and one piece of eye gaze management information with the “gazed objectID” of “C003”. In this case, the output unit 127 calculates, as “4”, thenumber of times the eye gaze of the user is oriented to the targetobject having the “gazed object ID” of “COO”. Then, the output unit 127sets the calculated number of times “4” as the number of times the eyegaze of the user is oriented to each of the four target eye gaze pointswith the “image capturing date and time” of “2019/5/17 13:33:13” to“2019/5/17 13:33:16” included in the target object with the “gazedobject ID” of “C001”.

Similarly, the output unit 127 calculates, as “1”, the number of timesthe eye gaze of the user is oriented to one target eye gaze point withthe “image capturing date and time” of “2019/5/17 13:33:20” included inthe target object with the “gazed object ID” of “C002”. Furthermore, theoutput unit 127 calculates, as “1”, the number of times the eye gaze ofthe user is oriented to one target eye gaze point with the “imagecapturing date and time” of “2019/5/17 13:33:22” included in the targetobject with the “gazed object ID” of “C003”.

Similarly, for the second eye gaze usage information, the output unit127 calculates, as “3”, the number of times the eye gaze of the user isoriented to the three target eye gaze points with the “image capturingdate and time” of “2019/5/17 13:33:17” to “2019/5/17 13:33:19” includedin the target object with the “gazed object ID” of “C004”. In addition,the output unit 127 calculates, as “1”, the number of times the eye gazeof the user is oriented to one target eye gaze point with the “imagecapturing date and time” of “2019/5/17 13:33:21” included in the targetobject with the “gazed object ID” of “C002”. In addition, the outputunit 127 calculates, as “I”, the number of times the eye gaze of theuser is oriented to one target eye gaze point with the “image capturingdate and time” of “2019/5/17 13:33:23” included in the target objectwith the “gazed object ID” of “C003”.

Next, for each of the first eye gaze usage information and the secondeye gaze usage information, the output unit 127 displays, on the displaydevice 300, the number of times the eye gaze of the user has beenoriented to each target eye gaze point in a region where the targetobject including each target eye gaze point included in each eye gazeusage information is displayed.

For example, on the display device 300, the output unit 127 displays“4”, which is the number of times the eye gaze of the user has beenoriented to each of the four target eye gaze points, in the region wherethe target object with the “gazed object ID” of “C001” is displayed,including the four target eye gaze points with the “image capturing dateand time” of “2019/5/17 13:33:13” to “2019/5/17 13:33:16” included inthe first eye gaze usage information.

Similarly, on the display device 300, the output unit 127 displays “1”,which is the number of times the eye gaze of the user has been orientedto one target eye gaze point, in the region where the target object withthe “gazed object ID” of “C002” is displayed, including the one targeteye gaze point with the “image capturing date and time” of “2019/5/1713:33:20” included in the first eye gaze usage information. In addition,on the display device 300, the output unit 127 displays “1”, which isthe number of times the eye gaze of the user has been oriented to onetarget eye gaze point, in the region where the target object with the“gazed object ID” of “C003” is displayed, including the one target eyegaze point with the “image capturing date and time” of “2019/5/1713:33:22” included in the first eye gaze usage information.

Similarly, on the display device 300, the output unit 127 displays “3”,which is the number of times the eye gaze of the user has been orientedto each of the three target eye gaze points, in the region where thetarget object with the “gazed object ID” of “C004” is displayed,including the three target eye gaze points with the “image capturingdate and time” of “2019/5/17 13:33:17” to “2019/5/17 13:33:19” includedin the second eye gaze usage information. In addition, on the displaydevice 300, the output unit 127 displays “1”, which is the number oftimes the eye gaze of the user has been oriented to one target eye gazepoint, in the region where the target object with the “gazed object ID”of “C002” is displayed, including the one target eye gaze point with the“image capturing date and time” of “2019/5/17 13:33:21” included in thesecond eye gaze usage information. In addition, on the display device300, the output unit 127 displays “1”, which is the number of times theeye gaze of the user has been oriented to one target eye gaze point, inthe region where the target object with the “gazed object ID” of “C003”is displayed, including the one target eye gaze point with the “imagecapturing date and time” of “2019/5/17 13:33:23” included in the secondeye gaze usage information.

Next, for each of the first eye gaze usage information and the secondeye gaze usage information, the output unit 127 refers to each targeteye gaze point included in each eye gaze usage information inchronological order of “image capturing date and time” corresponding toeach target eye gaze point. Then, the output unit 127 outputs a straightline connecting the currently referred target eye gaze point and thetarget eye gaze point to be referred to next to the display device 300as a movement route of the eye gaze of the user to the target eye gazepoint to be referred to next.

For example, the output unit 127 outputs, to the display device 300, astraight line connecting the target eye gaze point whose “imagecapturing date and time” is the oldest “2019/5/17 13:33:13” and thetarget eye gaze point whose “image capturing date and time” is the nextoldest “2019/5/17 13:33:14” among the target eye gaze points included inthe first eye gaze usage information. Similarly, the output unit 127outputs, to the display device 300, a straight line connecting thetarget eye gaze point whose “image capturing date and time” is“2019/5/17 13:33:14” and the target eye gaze point whose “imagecapturing date and time” is the next oldest “2019/5/17 13:33:15” amongthe target eye gaze points included in the first eye gaze usageinformation. Thereafter, similarly, the output unit 127 outputs thestraight line to the display device 300, and finally, outputs, to thedisplay device 300, a straight line connecting the target eye gaze pointwhose “image capturing date and time” is the newest “2019/5/17 13:33:22”and the target eye gaze point whose “image capturing date and time” isthe next newest “2019/5/17 13:33:20” among the target eye gaze pointsincluded in the first eye gaze usage information.

Similarly, the output unit 127 outputs, to the display device 300, astraight line connecting the target eye gaze point whose “imagecapturing date and time” is the oldest “2019/5/17 13:33:17” and thetarget eye gaze point whose “image capturing date and time” is the nextoldest “2019/5/17 13:33:18” among the target eye gaze points included inthe second eye gaze usage information. Thereafter, similarly, the outputunit 127 outputs the straight line to the display device 300, andfinally, outputs, to the display device 300, a straight line connectingthe target eye gaze point whose “image capturing date and time” is thenewest “2019/5/17 13:33:23” and the target eye gaze point whose “imagecapturing date and time” is the next newest “2019/5/17 13:33:21” amongthe target eye gaze points included in the second eye gaze usageinformation.

According to the present configuration, the viewer of the display device300 can easily grasp which position on which movement route the eye gazeof the user having the same attribute is oriented to many times.

Fifth Embodiment

When the number of users of the image processing system 1 becomes aslarge as, for example, several thousands, the number of records of theauthentication information stored in the authentication informationtable T1 (FIG. 4) increases. In this case, the number of recordsreferred to in the processing of the personal authentication using theiris information and the authentication information table T1 (FIG. 4) bythe iris authentication unit 123 in step S3 (FIG. 6) increases, and thetime required for the processing increases. As a result, start of theprocessing in and after step S4 (FIG. 6) is delayed, and there is apossibility that the eye gaze management information cannot be quicklygenerated.

In the fifth embodiment, in order to avoid such a problem, the irisauthentication unit 123 performs processing of the personalauthentication of the person 400 using detected iris information and theauthentication information storage unit 141, which is performed afterthe iris information is detected in step S3 (FIG. 6), at timingdifferent from the processing for detecting the eye gaze information.Then, the management information generation unit 126 acquires thepersonal information of the person 400 who has been subjected to thepersonal authentication after the processing of the personalauthentication, and generates the eye gaze management information inwhich the acquired personal information is associated with the eye gazeinformation detected at another timing. A method for generating eye gazemanagement information in the fifth embodiment will be described belowwith reference to FIGS. 9 to 11.

FIGS. 9 and 10 are flowcharts showing an example of the operation ofimage processing device 100 according to the fifth embodiment.Specifically, the operation of the image processing device 100 shown inFIG. 9 is started periodically (e.g., every second), similarly to theoperation of the image processing device 100 shown in FIG. 6. When theoperation of the image processing device 100 is started, steps S1 and S2described above are performed.

Next, similarly to step S3 (FIG. 6), the iris authentication unit 123detects iris information indicating the iris 56 of the eye of the person400 from the eye region 50 detected in step S2 (step S31). After stepS31, step S4 (FIG. 6) is omitted, and steps S5 and S6 are performed.

Next, the management information generation unit 126 generates temporaryeye gaze management information in which the iris information detectedin step S31 is associated with the eye gaze information detected in stepS6 (step S71). The output unit 127 stores the temporary eye gazemanagement information generated in step S71 into a temporary managementinformation table (step S81). The temporary management information tableis a table that stores the temporary eye gaze management informationregarding one or more persons 400 generated by the managementinformation generation unit 126. The temporary management informationtable is stored in a memory (not illustrated) included in the processor120 or a storage device (not illustrated) such as a hard disk drive or asolid state drive included in the image processing device 100.

FIG. 11 is a view showing an example of a temporary managementinformation table T4. For example, in step S71, as shown in FIG. 11, thetemporary management information table T4 stores temporary eye gazemanagement information in which “image capturing date and time”. “eyegaze position X coordinate”, “eye gaze position Y coordinate”, and“gazed object ID” included in the eye gaze information detected in stepS6 are associated with “iris data”, “pupil diameter size”, and “irisdiameter size” included in the iris information detected in step S31.The “iris data” is iris data included in the iris information detectedin step S31. The “pupil diameter size” is the length of the diameter ofthe outer edge of the pupil 55 included in the iris information detectedin step S31. The “iris diameter size” is the length of the diameter ofthe outer edge of the iris 56 included in the iris information detectedin step S31.

The operation of the image processing device 100 shown in FIG. 10 isstarted at an arbitrary timing when one or more pieces of temporary eyegaze management information items are stored in the temporary managementinformation table T4. When the operation of the image processing device100 shown in FIG. 10 is started, the iris authentication unit 123 refersto one piece of temporary eye gaze management information stored in thetemporary management information table T4, and performs the personalauthentication of the person 400 similarly to step S3 (FIG. 6) using theiris information included in the referred temporary eye gaze managementinformation (step S32). Next, the management information generation unit126 acquires the personal information of the person 400 who has beensubjected to the personal authentication in step S32 similarly to stepS4 (FIG. 6) (step S42).

Next, similarly to step S7 (FIG. 6), the management informationgeneration unit 126 generates the eye gaze management information inwhich the eye gaze information included in one piece of temporary eyegaze management information referred to in step S32 is associated withthe personal information acquired in step S42 (step S72). Next, themanagement information generation unit 126 deletes the one piece oftemporary eye gaze management information referred to in step S32 fromthe temporary management information table T4 (step S73). Next, theoutput unit 127 stores, similarly to step Sg (FIG. 6), the eye gazemanagement information generated in step S72 into the managementinformation table T3 (FIG. 7) (step S82).

According to the present configuration, the processing of personalauthentication, which is likely to increase the processing time, can beperformed at an arbitrary timing when one or more pieces of temporaryeye gaze management information are stored in the temporary managementinformation table T4. This makes it possible to eliminate a possibilitythat a large time difference occurs between the detection timing of eyegaze information used to generate eye gaze management information andthe acquisition timing of the personal information associated with theeye gaze information. Thus, the eye gaze management information can bequickly generated.

It is assumed that a difference between the acquisition date and time ofthe personal information in step S42 and the “image capturing date andtime” included in the eye gaze information associated with the personalinformation in step S72 is equal to or greater than a predeterminedtime. In this case, the acquired personal information is personalinformation stored in the user information table T2 (FIG. 5) at the timepoint when a time equal to or greater than the predetermined time haselapsed since the image data used to detect the eye gaze information wasacquired. Therefore, there is a possibility that the personalinformation is different from the personal information of the user atthe time point when the image data was acquired. Therefore, in a casewhere the difference between the acquisition date and time of thepersonal information in step S42 and the “image capturing date and time”included in the eye gaze information associated with the personalinformation in step S72 is equal to or greater than a predeterminedtime, the eye gaze management information may not be generated in stepS72.

Sixth Embodiment

In the sixth embodiment, the degree of interest of the person 400 isestimated. FIG. 12 is a block diagram showing an example of a detailedconfiguration of the image processing system IA according to the sixthembodiment. In the present embodiment, identical components as those inthe above-described embodiments are given identical reference numerals,and description thereof will be omitted. Furthermore, in FIG. 12, ablock having an identical name as that in FIG. 2 but having a differentfunction is given a reference sign A at the end.

A processor 120A further includes a degree of interest estimation unit128.

The degree of interest estimation unit 128 estimates the degree ofinterest of the person 400 by the following processing. First, thedegree of interest estimation unit 128 detects an eyebrow and a cornerof the mouth from the face region using the facial feature pointdetected by the facial feature detection unit 124. Here, the degree ofinterest estimation unit 128 is only required to detect the eyebrow andthe corner of the mouth by specifying the feature points to which thelandmark point numbers respectively corresponding to the eyebrow and thecorner of the mouth are imparted among the facial feature pointsdetected by the facial feature detection unit 124.

Next, the degree of interest estimation unit 128 estimates the degree ofinterest of the person 400 based on the eye gaze information detected bythe eye gaze detection unit 125 and the position of the eyebrow and theposition of the corner of the mouth having been detected, and outputsthe degree of interest to the display device 300. Specifically, thedegree of interest estimation unit 128 acquires, from a memory (notillustrated) for example, pattern data in which standard positions ofthe eyebrow and the corner of the mouth when a person puts on variousexpressions such as joy, surprise, anger, sadness, and blankness aredescribed in advance. Then, the degree of interest estimation unit 128collates the detected positions of the eyebrow and the corner of themouth of the person 400 with the pattern data, and estimates theexpression of the person 400. Then, using the estimated expression ofthe person 400 and the eye gaze indicated by the eye gaze information,the degree of interest estimation unit 128 specifies as to whatexpression the person 400 makes when the eye gaze of the person 400 isin which direction or the eye gaze point of the person 400 is present inwhich position. That is, the degree of interest estimation unit 128specifies, as the degree of interest of the person 400, data in whichthe eye gaze information and the expression of the person 400 areassociated with each other. Note that, here, the degree of interestestimation unit 128 is described here to estimate the degree of interestbased on the eyebrow and the corner of the mouth, but this is anexample, and the degree of interest may be estimated based on one of theeyebrow and the corner of the mouth.

As described above, according to the present embodiment, since thedegree of interest of the person 400 is estimated by further using theeyebrow and the corner of the mouth in addition to the eye gazeinformation, the degree of interest can be estimated with higheraccuracy as compared with the degree of interest estimation based onlyon the eye gaze information.

(Modifications)

(1) In the above-described embodiment, the case where the operation ofthe image processing device 100 shown in FIGS. 6 and 9 is startedperiodically (e.g., every second) has been described. However, insteadof this, the operation of the image processing device 100 shown in FIGS.6 and 9 may be started every time the image data of the face of theperson 400 is captured by the camera 200. Alternatively, the operationof the image processing device 100 shown in FIGS. 6 and 9 may be startedthe predetermined number of times every time the image data of the faceof the person 400 is captured a predetermined number of times by thecamera 200.

(2) If an infrared light camera is adopted as the camera 200, theinfrared light camera is only required to be an infrared light camerausing infrared light in a predetermined second wavelength band in whichthe spectral intensity of sunlight is attenuated more than apredetermined first wavelength. The predetermined first wavelength is,for example, 850 nm. The predetermined second wavelength is, forexample, 940 nm. The second wavelength band does not include, forexample, 850 nm and is a band having a predetermined width with 940 nmas a reference (e.g., the center). As an infrared light camera thatcaptures near-infrared light, one that uses infrared light of 850 nm isknown. However, since the spectral intensity of sunlight is notsufficiently attenuated at 850 nm, there is a possibility that highlyaccurate eye gaze information detection cannot be performed outdoorswhere the spectral intensity of sunlight is strong. Therefore, as aninfrared light camera, the present disclosure employs a camera that usesinfrared light in a band of 940 nm, for example. This makes it possibleto perform highly accurate eye gaze information detection even outdoorswhere the spectral intensity of sunlight is strong. Here, thepredetermined second wavelength is 940 nm, but this is an example, andmay be a wavelength slightly shifted from 940 nm. Note that the infraredlight camera using the infrared light of the second wavelength is, forexample, a camera including a light projector that irradiates with theinfrared light of the second wavelength.

(3) In the above embodiment, the eye gaze information is described toinclude the coordinate data indicating the eye gaze point, but thepresent disclosure is not limited thereto. For example, the eye gazeinformation may include coordinate data indicating an eye gaze planethat is a region having a predetermined shape (e.g., a circle, aquadrangle, or the like) with a predetermined size with the eye gazepoint as a reference (e.g., the center). This makes it possible toappropriately determine the eye gaze target object without depending onthe distance between the person 400 and the eye gaze target object orthe size of the eye gaze target object.

(4) in the above-described embodiment, an example in which the imageprocessing system 1 is applied to a digital signage system has beendescribed, but the image processing system 1 is also applicable to, forexample, an exhibition. In this case, assuming that the participant ofthe exhibition is a user of the image processing system 1, the workplace of the participant is only required to be included in theattribute information of the user stored in the user information tableT2. Furthermore, the eye gaze information is only required to includeexhibit information indicating an exhibit of the exhibition existing ata position to which the eye gaze of each user is oriented. The exhibitinformation may include, for example, the name of the exhibit and/or theidentifier of the exhibit. Then, similarly to the above-described thirdembodiment, the output unit 127 may display, on the display device 300,a heat map representing the relationship between the exhibit of theexhibition indicated by the exhibit information and the frequency atwhich the eye gaze of the user is oriented to the exhibit of theexhibition. In this case, the viewer of the heat map having been outputcan easily grasp, for example, in the exhibition, an eye gaze of aparticipant of which work place is highly frequently oriented to whichexhibit.

In addition, the attribute information of the user stored in the userinformation table T2 may include the job type of the participant of theexhibition, and the processing similar to that of the above-describedthird embodiment may be performed. In this case, the viewer of the heatmap output by the output unit 127 can easily grasp, in the exhibition,an eye gaze of a participant of which job type is highly frequentlyoriented to which exhibit.

Alternatively, the image processing system 1 can also be applied to, forexample, a manufacturing site. In this case, assuming that the worker atthe manufacturing site is a user of the image processing system 1, thework proficiency of the worker may be included in the attributeinformation of the user stored in the user information table T2. The eyegaze information is only required to include work target informationindicating a work target present at a position to which the eye gaze ofeach user is oriented. The work target information may include, forexample, a name of the work target and/or an identifier of the worktarget. Then, similarly to the third embodiment, the output unit 127 isonly required to display, on the display device 300, the heat maprepresenting the relationship between the work target indicated by thework target information and the frequency at which the eye gaze of theuser is oriented to the work target. In this case, the viewer of theheat map having been output can easily grasp, for example, at themanufacturing site, which work target an eye gaze of a highly proficientworker is highly frequently oriented to.

INDUSTRIAL APPLICABILITY

Since the present disclosure can accurately generate information inwhich personal information of a user is associated with informationindicating an eye gaze of the user with a simple configuration, thepresent disclosure is useful in estimation of an interest target of aperson using eye gaze information, state estimation of a person, a userinterface using eye gaze, and the like.

1. An information processing method in an information processing device,the information processing method comprising: for each of one or moreusers, acquiring image data including an eye of each of the users;detecting eye gaze information indicating an eye gaze of each of theusers based on information indicating the eye of each of the usersincluded in the image data; performing personal authentication on eachof the users based on information indicating the eye of each of theusers included in the image data; acquiring personal information foridentifying each of the users for which the personal authentication hasbeen performed; generating management information in which the personalinformation of the one or more users and the eye gaze information of theone or more users are associated with each other; and outputting themanagement information.
 2. The information processing method accordingto claim 1, wherein the personal information includes one or moreattributes indicating a nature or a feature of each of the users, and inoutput of the management information, based on the managementinformation, eye gaze usage information in which the eye gazeinformation is classified for each of the one or more attributes isfurther generated, and the eye gaze usage information is output.
 3. Theinformation processing method according to claim 2, wherein the one ormore attributes includes one or more of an age, a gender, a work place,and a job type.
 4. The information processing method according to claim2, wherein the eye gaze information includes eye gaze positioninformation indicating a position to which an eye gaze of each of theusers is oriented, and the eye gaze usage information is a heat maprepresenting a relationship between a position indicated by the eye gazeposition information and a frequency at which the eye gaze of the useris oriented to a position indicated by the eye gaze positioninformation.
 5. The information processing method according to claim 2,wherein the eye gaze information includes eye gaze position informationindicating a position to which an eye gaze of each of the users isoriented, and the eye gaze usage information is a gaze plot representinga relationship among the position indicated by the eye gaze positioninformation, a number of times the eye gaze of the user is oriented tothe position indicated by the eye gaze position information, and amovement route of the eye gaze of the user to the position indicated bythe eye gaze position information.
 6. The information processing methodaccording to claim 1, wherein in detection of the eye gaze information,information indicating the eye of each of the users and informationindicating the orientation of the face of each of the users are detectedfrom the image data, and the eye gaze information is detected based onthe detected information indicating the eye of each of the users and thedetected information indicating the orientation of the face of each ofthe users.
 7. The information processing method according to claim 1,wherein in personal authentication of each of the users, irisinformation indicating an iris of the eye of each of the users isdetected from the image data, and each of the users is subjected to thepersonal authentication based on the detected iris information.
 8. Theinformation processing method according to claim 2, wherein the one ormore users are participants in an exhibition, the one or more attributesinclude a work place of the participants, the eye gaze informationincludes exhibit information indicating an exhibit of the exhibitionexisting at a position to which an eye gaze of each of the users isoriented, and the eye gaze usage information is a heat map representinga relationship between an exhibit of the exhibition indicated by theexhibit information and a frequency at which the eye gaze of the user isoriented to the exhibit of the exhibition.
 9. The information processingmethod according to claim 2, wherein the one or more users are workersat a manufacturing site, the one or more attributes include workproficiency of the workers, the eye gaze information include work targetinformation indicating a work target present at a position to which aneye gaze of each of the users is oriented, and the eye gaze usageinformation is a heat map representing a relationship between the worktarget indicated by the work target information and a frequency at whichthe eye gaze of the user is oriented to the work target.
 10. Theinformation processing method according to claim 7, wherein the imagedata is captured by an infrared light camera.
 11. An informationprocessing device comprising: an image acquisition unit that acquires,for each of one or more users, image data including an eye of each ofthe users; an eye gaze detection unit that detects, for each of the oneor more users, eye gaze information indicating an eye gaze of each ofthe users based on information indicating an eye of each of the usersincluded in the image data; an authentication unit that performs, foreach of the one or more users, personal authentication on each of theusers based on information indicating an eye of each of the usersincluded in the image data; a personal information acquisition unit thatacquires, for each of the one or more users, personal information foridentifying each of the users for which the personal authentication hasbeen performed; a management information generation unit that generatesmanagement information in which the personal information of the one ormore users and the eye gaze information of the one or more users areassociated with each other; and an output unit that outputs themanagement information.
 12. A non-transitory computer readable storagemedium storing a control program of an information processing device,the control program causing a computer equipped with the informationprocessing device to function as an image acquisition unit thatacquires, for each of one or more users, image data including an eye ofeach of the users, an eye gaze detection unit that detects, for each ofthe one or more users, eye gaze information indicating an eye gaze ofeach of the users based on information indicating the eye of each of theusers included in the image data, an authentication unit that performs,for each of the one or more users, personal authentication on each ofthe users based on information indicating the eye of each of the usersincluded in the image data, a personal information acquisition unit thatacquires, for each of the one or more users, personal information foridentifying each of the users for which the personal authentication hasbeen performed, a management information generation unit that generatesmanagement information in which the personal information of the one ormore users and the eye gaze information of the one or more users areassociated with each other, and an output unit that outputs themanagement information.